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Basic  Assumption 


Representations  vary  in  how  effective 
they  are  at  facilitating: 

-  Shared  understanding 

-  Coordination 
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Overall  Objectives 


1 .  Develop  guidelines 

•  how  best  to  annotate  videos  and  video  annotate  stills  so  as  to  improve  shared 
understanding 

•  simulate  realistic  planning  contexts  in  order  to  determine  how  planners  and 
analysts  should  annotate  stills  and  videos  to  make  better  decisions 

2.  Experimentally  discover  media  factors  affecting: 

•  Shared  Understanding  -  explicit  and  operational  knowledge 

•  Decision-Making  -  what  information  format  best  helps  decision-makers 

•  Video-Augmented  Collaboration  -  is  collaboration  improved  by  using 
annotation  on  video  as  a  method  of  situating  problem  solving  and  discussion 

3.  Deepen  Theoretical  Framework 

•  Distributed  Cognition 

•  Knowing  That  vs.  Knowing  How 
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This  Year's  Objectives 


1 .  Develop  the  technical  environment 

•  Tools  to  annotate  videos  and  video  annotate  stills  off-line 

•  Tools  to  collaboratively  annotate  video  signal  in  real  time 

2.  Discover  which  factors  affect  single  subjects 

•  Shared  Understanding 

is  there  a  dissociation  between  explicit  and  operational  knowledge? 

•  Informed  Decision-Making  —  what  information  format  is  best  to 
pass  the  bubble  to  a  decision-maker 
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Theoretical  Orientation 


•  Distributed  Cognition 

-  Cognition  is  distributed  over  the  interactions  between  agents,  the 
resources  in  their  environment,  tools,  cultural  constraints  in  behavior  and 
tool/resource  use,  effectively  designed  environments,  etc 

-  Study  problem  solving  from  a  D-cog  perspective 

•  Representational  Efficiency: 

-  How  deeply  is  information  assimilated  -  1st  vs.  3rd  person  perspectives: 

-  3rd  person  knowledge:  world  state,  facts  and  figures 

-  1st  person  perspective:  operational,  perspectival: 

•  orienting  oneself  in  the  spatial  layout 

•  synchronizing  with  the  timing  of  actions 

•  internalizing  player  expectations  and  goal  structure  . . . 
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Long-term  Goals 


1 .  Improving  collaboration  and  decision  making 

-  Guidelines:  when  should  augmented  video  be  used  and 
when  is  it  not  worth  the  extra  cost 

-  Articles  &  Theoretical  models 

•  Extend  theory  of  distributed  cognition 

•  Coordination  Theory 

2.  Cognitive  efficiency  of  augmented  video  and  other 
representations 

-  Articles  &  Theoretical  models 

•  Extend  theory  of  representation  to  dynamic  representations 
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Definition  of  Key  Terms 


•  Passing  the  bubble:  communicating  situational  knowledge,  for 
example  when  a  commander  replaces  another  on  watch. 

•  Augmented  Video:  video  with  annotations 

•  Representational  Efficiency:  how  effective  different 

representational  formats  are  at  causing  a  subject  to  enter  a  specific  knowledge 
state 

•  Attention  Management:  a  method  for  controlling  what  an 

audience  focuses  on 

•  Cognitive  Load:  a  measure  of  how  much  of  a  subject’s  cognitive 
resources  are  recruited  in  a  task  or  activity 

•  Dynamic  Representations:  representations  that  change  over 

time  (animations,  video,  . . .) 
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The  game:  Starcraft 


•  A  logistics  and  strategy  real-time  game: 

-  Goal:  wipe  out  the  enemy! 

-  Three  interstellar  “races”:  Humans,  Protoss  and  Zerg  with  different  forces  and  weaknesses, 
different  units,  technologies,  etc. 

-  Strategy:  build  up  military,  technology  and  economy  to  pay  for  it. 

-  Strategic  choices: 

•  fortifications  versus  military  units, 

•  Small  high-tech  army  versus  large  low-tech  army 

-  Two  types  of  resources  must  be  gathered  from  different  sites  on  the  map,  by  the  relevant 
industrial  buildings  and  units.  Military  and  technological  buildings  allow  to  build  forces,  used 
to  defeat  the  enemy. 

•  Video  intro  to  Starcraft:  50  sec.  4  min.  4.5  min 

•  Strategic  expertise:  takes  500  to  1000  hours  to  achieve 

•  Game  duration:  about  40  minutes 
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Problem  Being  Addressed: 
Specific  Task  Domain 


The  role  of  augmented  video  in  passing  the  bubble: 


•  Which  types  of  augmented 
video  cause  cognitive 
overload 

•  Is  augmented  video  always 
better  than  well-chosen 
stills? 

•  Can  we  dissociate 
communicating  context  from 

communicating  intent  (what 
the  situation  is  from  what  we  want 
the  situation  to  be,  and  the  means 
to  get  there)? 
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Our  Current  Testbed 


•  Simplified  environment:  a  strategy  game 

•  The  bubble  is  passed  by  different  representation  types 

•  Several  measures  of  cognitive  efficiency: 


-  Explicit  knowledge  gained  (third  person,  “knowing  that”)  -  from 
questionnaire  (“what  units  is  my  enemy  using”) 

-  Task  performance  (Win/Lose,  time  to  completion,  Game  Score 
increment  in  first  5  minutes) 

-  Subjective  judgement  -  questionnaire  (overall  preference,  how 
informative,  “how  confident  do  you  feel  you  know  what’s  going 
on”) 
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Experimental  Design 


•  Within  subject  design  (each  subject  tries  all  the  stimuli,  and  is 
analyzed  independently) 

•  There  are  9  types  of  presentations,  with  variations  in  the  type  of 
annotation  and  the  type  of  background  that  bears  the  annotations 

•  For  each  type  we  plan  10  stimuli 

•  Every  time,  the  subject: 

-  Attends  the  presentation 

-  Is  tested  by  a  questionnaire,  to  know  what  has  been  learned  explicitly, 

-  Resumes  the  game,  against  the  original  enemy  player 

•  Implicit  and  context  understanding  is  tested  by  how  well  the  subject 
does  in  the  resumed  game. 
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Technical  Approach 


Types  of  Stimuli 


Random 

Stills/Control 

Chosen  Stills 

Video 

No 

Annotation 

Random  Stills,  Voice 

No  Annotations 

Chosen  Stills,  Voice 

No  Annotations 

Video,  Voice 

No  Annotations 

Static 

Annotation 

Random  Stills,  Voice 

Static  Annotations 

Chosen  Stills,  Voice 

Static  Annotations 

Video,  Voice 

Static  Annotations 

Dynamic 

Annotation 

Random  Stills,  Voice 
Dynamic  Annotations 

Chosen  Stills,  Voice 
Dynamic  Annotations 

Video,  Voice 

Dynamic  Annotations 
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Stimulus  Demo 


SELECTED  STILLS, 
NO  ANNOTATION 


^J 


(if  it  is  not  playing  after  4  seconds,  double-click  here  to  start  presentation  in  media  player) 
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Current  Progress 


•  Start  date:  April  2002 

•  Experimental  progress 

•  70  Stimuli  prepared 

•  46  stimuli  ran  on  2  subjects 

•  Technical  Progress 

•  Constructed  annotation  environment,  with  an  option  for 
collaborative  annotation 
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1 .  Video  Annotation  Environment 

100%  complete 

2.  Creation  of  Experimental  Stimuli 

70%  complete 

3.  Subjects  attend  the  Stimuli  and 
resume  the  game 

25%  complete 

4.  Collaborative  Annotation 
Environment 

60%  complete 
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Results  -  Metrics 


•  Explicit  knowledge 

(twelve  questions  about  items  in  the  presentation) 

•  Subjective  Quality  of  Stimulus 

(overall  rating,  how  confident,  how  informative) 

•  Effectiveness 

(gain/loss)*(l/  time  to  completion) 

•  Improvement  in  5  minutes 

(gain  in  percentage  of  total  score  in  the  first  five  minutes) 
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What  is  “effectiveness”? 
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Video  Annotation  Chain 
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Recommendations  /  Lessons  learned 


•  Tools: 

-  Capture,  basic  production,  annotation  of  stills 
(including  dynamic  annotations):  Camtasia 

-  Dynamic  annotation  of  video,  off-line:  Premiere  + 
AfterEffects  +  Vector  Paint 

-  Dynamic  annotation  of  video,  on-line:  Pointmaker 

•  Experimental  Protocol:  record  everything!  (for 

example,  subjects  talk  about  their  games  afterwards,  revealing  the  structure  of 
their  beliefs  about  the  game,  how  it  induced  bright  moves  or  mistakes,  and 
how  it  changed  during  the  game) 
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Planned  Publications 


•  Article  on  The  Representational  Efficiency 
of  Augmented  Video  and  Dynamic 
Representations 

•  Article  on  Augmented  Collaboration:  using 
Augmented  Video  as  a  Method  of  Situating 
Problem  Solving 
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